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ABSTRACT 

The so-called "symplectic method" is used for studying the linear stability of a self- 
gravitating collisionless stellar system, in which the particles are also submitted to an 
external potential. The system is steady and spherically symmetric, and its distribution 
^\ function /o thus depends only on the energy E and the squarred angular momentum 

of a particle. Assuming that dfo/dE < 0, it is first shown that stability holds with 
y—i respect to all the spherical perturbations - a statement which turns out to be also 

^ valid for a rotating spherical system. Thus it is proven that the energy of an arbitrary 

^ aspherical perturbation associated to a "preserving generator" 6gi [i.e., one satisfying 

dfo/dL^{6gi, L^} = 0] is always positive if dfo/dL^ < and the external mass density 
is a decreasing function of the distance r to the center. This implies in particular 
04 (under the latter condition) the stability of an isotropic system with respect to all the 

04 perturbations. 

Some new remarks on the relation between the symmetry of the system and the 
(T^ form of /o are also reported. It is argued in particular that a system with a distribution 

function of the form /o = fo(E, L^) is necessarily spherically symmetric. 
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INTRODUCTION 



^ '[T^p analytical study of the linear stability of a steady collisionless stellar system is a quite formidable task which has progressed 
Q with great difficulties since the pioneering work of Antonov ((Antonov 1962), (Antonov 1973)). The investigation methods 
» I which have been used up to now fall into two categories: The normal mode approach, and the energetic approach. The first one 
"^is presented in a systematic way in the most recent monograph by Palmer ((Palmer 1994), see also (Fridman & Polyachenko 
1984) and references in these two books). It consists to derive from the linearized equations of motion (Vlasov-Poisson's 
system, VP hereafter) a dispersion relation for the eigenfrequencies of the system, and to try to extract from it as much 
information as possible on the nature of these numbers - an instability being present if at least one of them has a nonzero 
imaginary part, while stability holds if they are all real. The second method consists in constructing a quadratic functional 
W over a set Q of admissible test functions satisfying some of the constraints actually fulfilled by any solution of VP. The 
system is found to be stable if W keeps the same sign over Q. Actually, there are several types of energy methods, which 
correspond to different choices for W and Q. The simplest one is the "thermodynamic method" ((Lynden-Bell & Sannit 1969), 
(Ipser 1974), (Ipser & Horwitz 1979)), in which only a very few constraints are taken into account. On the contrary, all the 
VP constraints are retained in the more elaborate "energy principle" ((Antonov 1962),(Kulsrud & Mark 1970), (Kandrup & 
Sygnet 1985)), as well as in the "symplectic method" ((Bartholomew 1971), (Kandrup 1990), (Kandrup 1991), (Perez 1995) 
and (Perez & Aly 1995), Paper I hereafter), the latter differing from the former by a restriction imposed to the perturbations, 
which are taken to be "symplectic" - i.e., they are generated by the infinitesimal canonical transforms acting on the steady 
state. 

Our aim here is to reinvestigate the linear stability of a spherically symmetric system by fully working in the symplectic 
framework, with the hope of extending - or at least clarifying - some of the results which have been obtained thus far by the 
other energy methods. Our paper thus presents an interesting illustration of the general principles discussed in Paper I. At 
the same time, it also provides the theoretical background for the following paper of this series (Perez et al. 1995), in which 
the problem is tackled from a numerical point of view. It should be stressed from the very beginning that the method which is 
used here has not proven thus far to be able to reproduce all the results based on a normal mode analysis reported in (Palmer 
1994). The latter, however, are sometimes based on approximations which appear quite difficult to justify rigorously, while 
ours are derived without the help of any approximation. 

Our plan is as follows. In §2, we state precisely our assumptions - which include the possible presence of an external 
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potential, generally not considered by previous authors -, and we develop some new considerations on the relations between 
the symmetry of the steady states we are interested in and the form of their distribution functions /o. In §3, we recall the 
definition of a linear symplectic perturbation, the expression of its energy (which we cast into a particularly useful form), and 
the relations of the latter with the stability properties of the system. Thus we consider in details stability with respect to 
spherical (§4) and aspherical (§5) perturbations, respectively, assuming that /o is a decreasing function of the energy. Some 
extensions of our results to spherical rotating systems (Lynden-Bell 1960), and to systems with nonmonotonic /o, are reported 
in §6. 



2 EQUILIBRIUM 

2.1 Notations and equations 

We consider a system constituted of a large number of gravitating particles interacting together, and also submitted to an 
"external" potential $e. We denote as x and v, respectively, the position and the velocity of a particle with respect to a Galilean 
frame (0;x,y,z), and as { := (x, v) the corresponding point in the phase space Most often, we shall represent { either 
by its standard spherical coordinates (r,d,4>,Vr,vg,v^) (with 9 measured from the 2-axis), or by its "canonical" coordinates 
(r, 9, 4>,pr,pe,P4,), with pr = Vr,pe = rvg,p^ = r sin 9v^ being the "conjugated" momenta of (r, 9, 4>) (e.g., (Landau & Lifchitz 
1966)). 

We assume that: 

a) The external potential is spherically symmetric about O, i.e., $e = ^e{r). In most practical applications, $e will 
represent the potential either of a central massive object or of a massive halo. For our theoretical purpose, however, we do 
not need to fix precisely its form, and we just consider it as being created by some matter distributed with the given smooth 
density Pe{r). The total respective masses Me of that distribution and M of the system, are finite. 

b) The statistical state of the system is described at each time t by a one-particle distribution function T(S^,t,m), with 
.?-'({, t, m) dfdm representing the number of particles of mass between m and m + dm contained in the phase space volume 
d{ around 

c) Collisions between the particles are negligible. Then it is sufficient to consider the mass-averaged distribution function 



f{^,t):= J ■mT{^,t,m)dm. (1) 
/ solves the VP system 

^ + v.V,/-V,*.V./=^ + {/,/*} = 0, (2) 

V^* = 47rG(p + pe) :=47rG (^y"/dv + pe^ , (3) 

*=r^oo 0(r"^) and lim /(x, v) = , (4) 

( r,^J ) ^ CO 

with the decrease of / at infinity being supposed to be sufficiently fast. Here, ^' := $ -|- $e is the total gravitational potential 
created by the particles and the external masses, 

Mx,v,t):=y + *(x,t) (5) 
is the one-particle Hamiltonian, and {., .} denotes the Poisson bracket, defined by 

{/i,/2} := V,/i.V./2 - V./i.V,/2. (6) 

Note that there is no explicit dependance on the mass of the particles appearing in Eqs. (2)-(3). 

Instead of (2)-(3), we may use only Eq. (2) in which we have substituted for $ the unique solution to Eqs. (3)-(4). The 
latter can be written in integral form as $ = K[f], with the operator K being quite generally defined by 

/a/](x) := -G I J% di'. (7) 



2.2 Spherically symmetric steady states 

We are interested here in a steady state which is spherically symmetric with respect to O, and then, in particular, occupies 
a sphere of radius R (< oo) in the physical space. Hence fo - assumed from now on to be a sufficiently smooth function - 
and the associated potential $o = -R'[/o] are left unchanged by any rotation of R'^ of center O, which implies that they can 
depend only on the three geometrical invariants r, v and x.v = rvr - i.e., they are necessarily of the form fo = fo{r,v,rVr) 
and $0 = ^o{r), respectively. 

Let us introduce the two quantities (we set Vt := v — Vri) 
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E:=ho:=^ + Mr) ^= f + f + Mr) + $e(r) (8) 
and 

j-2 2 2 2 / 2 , 2 ^ 2 , P 4> t n\ 

sm fc' 

which represent the energy of a particle in the potential ^'o and the squarred modulus of its angular momentum with respect 
to O. Clearly, we can express v in terms of r, Vr and , and thus write 

fo = fo{r,Vr,L^). (10) 
Injecting this expression into the time-independant version of Eq. (2), we obtain 

fr^+ — -*0^=0. (11) 

ar J avr 

From the theory of first order partial differential equations, we know that the general solution of Eq. (11) is a function of two 
independant integrals of its characteristic system 

^ = ^ (12) 

Two such integrals are obvious here: (as di^ does not appear), and E{r, Vr,L^). Thus we can conclude at once that 

fo = fo(E,L'). (13) 

The particular case fo = fo{E) corresponds to an isotropic equilibrium. 

The general form (13) of fo is well known, but it is usually derived (e.g., (Dejonghe 1986), (Binney & Tremaine 1987), 
(Palmer 1994)) by appealing to the so-called strong Jeans theorem, which deals with the isolating integrals of motion. As 
this theorem is far from being easy both to derive and to apply, we felt that it was a useful task to provide the reader with a 
completely straigthforward proof. In particular, the proof presented here shows most clearly the basic reasons why fo depends 
on only two variables: Symmetry requires fo to depends on only three variables, and the steady VP reduces this number by 
one unit. 

Evidently, the potential ^'o appearing in i? is a solution of 



^2 -r Id 



(r^^) = 47rG(po + Pe) := 47rG (^J fo(E, i') dv + fj}j , (14) 

*o =r^oo 0(r-'). (15) 

An immediate consequence (which will prove quite important hereafter) of these equations is that ^'o(j') is a strictly increasing 
negative function (as p > and > 0). $o and $e (if Pe ^ 0) have obviously the same property. 

The characteristics of the steady state VP describe the motion of the individual particles in the (self-consistent) potential 
^'o. In particular, if we substitute dr/dt for Vr, Eq. (12) and its integral E determine the radial motion of a particle of 
energy E and angular momentum modulus L, which can be described as a one-dimensional motion in the effective potential 
^eff '■= ^0 + i^/2r^. When r increases from to -|-oo, the latter decreases from -\-oo to a negative minimum, reached for 
ro(i^), and then increases to . As the system is confined inside the sphere of radius R and v^. > 0, we have necessarily 

E-{L^) := *e//[ro(i')] = *o[ro(i')] + ^[^^^^ <E< *o(i?) + ^ =: < (16) 

[with E'^(L^) = if i? = oo; for an isotropic state, we can even take E'^{L^) = ^o{R), a,s Vr = implies v = 0] - the 
particle thus bouncing between the two solutions r~ [E , L^) and r'^ [E , L^) \r~ < ro < r"*" < i?] to the equation ^eff{r) = E. 
Eq. (16) imposes to be bounded from above by the smallest solution L^ax of the equation E~{L^) = E'^(L^), or 
equivalently of ro(imax) = R [the existence of that solution results immediately from the monotonic increase of ro(i^) *]. 
Then we have eventually reached the conclusion that fo{E, L^) must vanish outside the energy range [E~{L^), E'^ {L^)'\, where 

^ i ^ Lmax- 

Hereafter, we shall denote as Qo the region of the phase space where fo > 0, and set 



dfo , , df 



foE ■■= — and foL2 := . (17) 



* This last statement is quoted very often in the literature, but without proof. To establish it, we remark that ^'^-^-^ = implies 
G(Mr + Mer) = E^ I T , where (Mr -I- Mer) denotes the total mass in a sphere of radius r, and we have used the integral of Eq. (14) 
(Newton's theorem); as the LHS of that equation is a nondecreasing function of r , while its RHS is a decreasing one, it has one and only 
one solution ro (i'^)- The conclusion thus follows by noting the obvious behaviour of ^e// when r ^ and r ^ oo. It is also interesting 
to note that ro(-£/^) increases monotonicaUy with (dro/dL^ > 0) from to oo, with ro oc L^/^ when 0, and ro oc when 

L2 ^ oo. 
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Unless otherwise specified, it will be assumed that 

foE < (18) 
in Qo- 

2.3 Symmetry of fo: Converse theorems 

In the previous subsection, we have shown that a steady spherically symmetric system has an fo of the form (13). It is then 
natural to address the converse question: // a steady state is characterized by a distribution function of the form fo{E, L^), is 
it necessarily spherically symmetric? 

Let us then assume that we have some steady state described by a sufficiently smooth distribution function fo{E,L^) 
[condition (18) is not imposed here]. Hence, fo does satisfy 

{fo,E} = foE{E,E} + foL2{L\E} = -2/oi2Vt.V*o(x) = 0, (19) 
while u := — ^'o(x) is a positive solution (with no a priori prescribed symmetry) to the nonlinear boundary value problem 

-Vlu (x) = 4tG fo (^y - «(x), r%?^ dv + pe(r)| =: ff[«(x), r] , (20) 

M =,^oo 0(r-'). (21) 

The function g defined in Eq. (20) is supposed to be sufficiently regular. Note that we have distinguished in the dependance 
of on X a part which arises from the dependance of fo on E, and a part which is related to both the dependances of fo on 
and of pe on r (this separation being clearly possible without any ambiguity). 

Consider first the case where /oi,2 ^ 0. Then Eq. (19) implies at once that ^'o(x) = ^'o(j') - i.e., spherical symmetry 
holds indeed - if, whenever /9o(x) > 0, there is a v such that /oj,2(x, v) ^ [this situation is certainly met if /oi,2 7^ in Qo - 
except maybe on a negligible set of points -, which happens, for instance, if fo is of one of the standard forms fo{E ± /2ra), 
fi{E)L^" , ... (Binney & Tremaine 1987)]. If the condition above is not satisfied, Eq. (19) only implies that ^'o(x) = ^o{r) 
inside some spherical shells, but the general conclusion can still be reached by applying a continuation argument to Eq. (20) 
(note that both ^'o and its normal derivative dr^o take constant values on the boundary of the shells). 

The situation appears to be more intricate when fo does not depend explicitly on {foL^ = 0)- But fortunately, we may 
then use some recent mathematical results which apply to the solutions to Eqs. (20)-(21) under some technical assumptions 
(Gidas et al. 1981) (the interested reader is referred to that paper for complete statements and proofs of the theorems we 
shall use). Let us first assume that = and g{u) = Oi^u") near m = 0, whith a > 4 [the last condition is satisfied in most 
cases of practical interest; for instance, it holds true when fo{E) = for E'^ < E, with the constant E'^ < - in which case 
the system is contained in a bounded volume -, and for Plummer's model, where g{u) oc (Binney & Tremaine 1987)]. Then 
we are in the conditions of Theorem 1 of (Gidas et al. 1981), and we can assert at once that u, and then ^'o, is necessarily 
spherically symmetric about the origin O (possibly after a translation). The fact that da/dr = — d^'o/dr < for r > 0, also 
asserted by the theorem, results more directly here from g being nonnegative (see §2.2). If pe ^ 0, symmetry around O is 
obtained as a consequence of theorem 1" of (Gidas et al. 1981) (after some slight changes made possible by the particular 
form of our g); we just need to assume that p'^ < 0, poPe 7^ at some point, and g(u, r) < cu°' near m = 0, with c > and 
a > 4. 

We have thus reached eventually the conclusion that, under reasonable assumptions, a state characterized by fo{E,L^) 
is spherically symmetric indeed. 



3 SYMPLECTIC ENERGY FUNCTIONAL AND LINEAR STABILITY 
3.1 Symplectic perturbations 

Let us now recall some of the basic facts about the symplectic approach to the stability of a steady state, referring the 
readers to (Bartholomew 1971), (Kandrup 1990), (Kandrup 1991) and Paper 1 for details. A linear "symplectic" perturbation 
is defined to be a first order change of the distribution function which is of the form 

fi = -{fo,gi}, (22) 

for some arbitrary regular phase-space function gi - the so-called "generator" -, that it is convenient to consider formally as 
being complex valued. Clearly, a symplectic perturbation admits of an infinity of generators, two of them differing by some 
function g commuting with fo {{fo,g} = 0). We shall say that gi is nontrivial if /i ^ 0. 
To any gi are also associated a first order variation of the potential. 



$1 = A'[/i] = A'[-{/o,ffi}] 



(23) 
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and a variation of the total energy which turns out to be second order in gi, being given by 

H^'\9i] = -l j{fo,gi}{E,gl}A^-^ J |V$i|^dx (24) 

("*" denotes complex conjugation). It is worth noticing that: 

a) H^^\gi] is a conserved quantity if gi is taken to be a solution of the linearized equation of motion 

^ = CM := {E, g^} + K[{fo, gi}] (25) 

[in which case fi = —{fo,gi} solves the linearized version of VP]. 

b) If the system is translated as a whole by an infinitesimal vector a (gi = a.v and /i = — a.V/o), then H^^\gi] = in 
the absence of external potential (this is intuitively obvious, and easy to check formally). This type of "neutral" displacements 
being of no interest, we shall eliminate them by imposing the admissible generators, whose set will be denoted by Q, to satisfy 



/ixd{ = J 5,iV,/od{ = (26) 
in the case where $e = - this condition impeding the center of mass of the system to be moved by the perturbation. 



3.2 Linear stability criteria 

The stability properties of the steady state turn out to be strongly related to the values taken by H^^^ on Q, and it is always 
an important first step when considering stability to classify as precisely as possible the generators gi according to the sign 
of that quantity. In fact, this study may even provide in some cases useful sufficient conditions of stability ((Bartholomew 
f97f), (Kandrup f990), (Kandrup f99f) and Paper 1): 

(a) If 

H^^\gi] > (or < 0) (27) 

for any nontrivial gi oft/, then the system is linearly stable with respect to any perturbation. Unfortunately, this condition may 
be possibly satisfied only if fo = fo{E) and foE < (actually, it may be fulfilled for a more general form of /o, corresponding 
to a uniformly rotating system). Indeed, if fo depends explicitly on some other integral, there does always exist both positive 
and negative energy perturbations [Paper 1. The main step in proving this result is the construction of two phase space 
functions a and /? and of a number e such that the first term in the RHS of Eq. (24) be positive (resp., negative) for the 
generator gi := ae'^^^]. 

(b) Suppose that we cannot prove Eq. (27) for all the admissible gi , but that we can identify in t/ a linear subspace t/' 
that is "closed" under evolution - which means that, if an arbitrary gi G t/' is taken as an initial condition for an evolution 
governed by Eq. (25), then the solution gi(t) stays in t/' forever]. Then the system is stable with respect to the corresponding 
restricted class of perturbations if Eq. (27) holds true for all the nontrivial elements of t/'. 



3.3 Decomposition of H'^^^ 

For our spherically symmetric system, it will prove convenient to transform the expression for -ff(^)[(/i] int o a different form. 
For that, we first introduce the following notations associated to the arbitrary phase function g and the arbitrary rotation TZ 
of about O: 

gizii) = g(Tlx,Tlv) , (28) 

H0 = ^ j Mi)dTl, (29) 

8g = g-g. (30) 

Clearly, the average 'g is spherically symmetric, i.e., 'g{TZ£,) = 'g{£.) for any TZ, and 'g = 'g{r, v, rvr). We have also the useful 
simple relations 

^ = i/ ^^^^'^ d7^ = 0, (3f) 
and, for any g and g' 

{9^,9'nm = {9,9'}im- (32) 

For a perturbation generated by gi , we can thus write 

91 ='9T+Sgi, (33) 

fi=Ti+Sfi, (34) 
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$1 = $1 + «$i . (35) 

Using the symmetry of /o and E, the rotational invariance of the Laplacian and the properties of the averaging process quoted 
above, we obtain immediately 

fi = -{fo,Ii} = -foE{E,J[} and */i = -{/o, } , (36) 
¥7= A'[-{/o,F}] and «$i = A-[-{/o,«ffi}], (37) 
and after some straightforward algebra, 

i7«[3i] = i7«[^] + i7«[«3i], (38) 
with of course 

H^'\9l] = \ J(-foE)\{E,gT}f di-^ J IV.ir^dx, (39) 

H^'\^9i] = -l j{fo,Sgi}{E,8gt}di-^ j |V,«$i|'dx. (40) 
We now consider in turn each of the two terms appearing in the RHS of Eq. (38). 



4 STABILITY WITH RESPECT TO SPHERICAL PERTURBATIONS 

4.1 A further decomposition 

For studying the term H^^^\0\, it is useful to effect one further decomposition by proceeding as follows. For any average 'g, 
we denote as 'g'^^~ , respectively, its symmetric and antisymmetric parts (with respect to the transform Vr —fr), defined by 

'g{r, V, rvr) = ''"'■) + ~i'Vr)] + ^tH''' ''"r) — 'gi'T, V, —rvr)] =: 'g^ (r, v, rvr) + 'g~ (r, v, rvr) . (41) 

With 

gT = gT^ +gT~ , (42) 

Ti=Ti^+Ti~, (43) 

we thus have 

Ti^ = -{fo,gT^} , (44) 
^= K[-{fo,gT-}], (45) 
and we get after some simple algebra, 

i7«[^] = i7«[^+] + i7«[^-], (46) 
with 

H^'\9T^] = \j (-foE)\{E,gI^}\' d{, (47) 

H^'\9T-] = \ j(-foE)\{E,gT-}f di-^ j |V.$rrdx. (48) 

The first part -ff(^)[F"^] of -ff(^)[F] is clearly nonnegative, and thus we just need to consider in details the second part 
H('\gT-]. 

4.2 The sign of H'^^^gT'] 

To study the sign of H^^^[gi~], we closely follow the method of (Sygnet et al. 1984), which originates in the work of (Gillon 
et al. 1976). However, in addition to the facts that we have an external potential $e and that we do not use (r,L^,E) as 
independant variables, there are some essential differences between their calculations and ours, that we shall discuss below. 

First, we transform the expression of -ff(^)[F"] int o a more convenient form. For that, we integrate once Poisson's 
equation 

2— 1 d 2d$r -, ,_^1 d / 2 



^-^1 = ^1-'' = -^""^ / {/o.F"} dv = 47rG-— / foEVrgT' dv , (49) 

J.Z cij. cij. J dr \ J I 

(where we have effected two integrations by part in the second member to get the RHS), which gives 



V$i = = ( 47rG / foEVrgi dv ) f . 
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Injecting this result into the second term of the RHS of Eq. (39) and using Schwartz's inequality, we obtain 
J |V:r$i|^ dx = 27rG J J foEVrgT' dv dx 

< 27rG J (^J(-foE)vUv^ (^J(-foE)\gT-f dv^ dx. 

Noticing that 

Po = y /o dv = J (-foE)vl dv 

(just remark that dv = 27r?)t d?)rd?)t and integrate by part with respect to Vr), we thus obtain 
H^'HaT} >\j i-foE) {\{E,gT-}\' -47rGpo\gT-\') d{ . 
We now make the change of variable 



(51) 



(52) 



(53) 



(54) 



whith the new function fi being still regular for rvr = as ^~(r, v, rvr = 0) = owing to the antisymmetry of Using the 
derivative property of the Poisson bracket, and the fact that the integral over the phase space of a Poisson bracket vanishes, 
we can rewrite Eq. (53) in the form 

H^^\gT-] > i j(-foE)[(rVrf\{E,ii}f+rVr{E,rVr}{E,\iif} + \lif{E,rVrf-47rGpo(rVr\li\f] d{ 

= I j i-f°E) [{TVrf\{E, + {E, \^l\\vr{E, TVr}} " \^l\\vr{E, {E , TVr}} 

— \tif{E, rvrY + TVrY — 4:-K G po\r V r tif\ di 

= \j i-foE) [(rVrf\{E, - {rVr{E, {E, rVr}} + 47rGpo(rVrf)] d{ . 
By a straightforward calculation, we obtain 

{E, {E, rVr}} = -rVr i^-^ + -— j = -rVr (47rG(po + Pe) + 
whence, injecting this expression into Eq. (55), 

H'^"'>[9T-]>1 Ji-foE){rvrf + (47rGpe + i^)] d{ 

the last inequality sign resulting from the negativeness of foE and the positiveness of pe and d^'o/dr. 

Let us now assume that H^^^[gi] = 0. Then H^^^[gi'^] = 0, whence {fo,'gi^} = 0; and H^^^[gi~] = 0, whence foE'gi~ = 0, 
and {fo,'gi~} = 0. Therefore fi = 0, and we can eventually conclude that H^^^[gi] > for any gi such that fi = —{fo,'gi} ^ 0. 

We can summarize the results of this section in the form of a theorem: 

Theorem 1: If the steady state satisfies condition (18), then H^^^\0\ > for any gi with a nontrivial average 'gi. 



1 d*o 
r dr 



> 0, 



(55) 



(56) 



(57) 



4.3 Consequences for stability 

It is quite obvious that the set Q' of the generators underlying the spherically symmetric perturbations is "closed with respect 
to evolution" in the sense of §3.2 - i.e., the solution gi{t) to Eq. (25) generates a spherically symmetric perturbations if the 
initial condition gi{(i) does. Then, by the general statement (b) of §3.2, we can conclude at once that any spherical steady 
state satisfying (18) is linearly stable with respect to all the spherical perturbations. 



4.4 Comparison with previous works 

Of course, this result has been reported many times in the literature for the case where $e = 0, and it is then worth explaining 
in a few words the differences between our proof and previous ones. In fact, in the framework of the energy methods, the 
game consists in any case to show the positiveness of some quadratic functional W[q\ over a set Q of admissible functions q. 

a) In a first class of methods ((Antonov 1962), (Kulsrud & Mark 1970), (Kandrup & Sygnet 1985)), Qi is taken to be 
the set of all the functions which vanish outside Qo and are antisymmetric in the change Vr —Vr, to which belongs the 
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antisymmetric part /i of any solution to the linearized VP. The relevant functional Wi may be derived in a standard way 
(e.g., (Laval et al. 1965)) from the equation of motion (second order in time!) 

^ -MK- (58) 



-foE dt^ 

satisfied by /i , the essential point being here the self-adjoint character (with respect to the usual hermitian product) of the 
operator M (Kandrup & Sygnet 1985). It turns out that 

m[^"] = i/(')[^"]. (59) 

It appears that the "energy principle" method and the symplectic one lead to formally identical functionals in the situation 
we are considering here. They can thus be treated by the similar technics, and the same results are obtained. However, the 
two methods are conceptually very different ((Bartholomew 1971), (Kandrup 1990), (Kandrup 1991), Paper 1). In the former, 
the derived "energy functional" is closely related to the peculiar properties of the second order equation one starts with [the 
existence of Wi results from the self-adjointness of M, which is just a consequence of the strong symmetries of the system], and 
it has no obvious physical meaning. The latter, on the contrary, is more transparent. It only deals with a first order equation 
- the one satisfied by the generators gi of the symplectic perturbations (which form a restricted class of perturbations) -, 
and H^^^\0\ is easily seen to be actually the energy of the perturbation - an interpretation which keeps its validity when one 
considers more general systems. 

b) In a second class of methods ((Lynden-Bell & Sannit 1969), (Ipser 1974), (Ipser & Horwitz 1979)), the relevant 
functional W2 is the second variation of E[f] + Hc[f]- Here, E[f] is the energy of / and 

Hc[f]:=- J C[f,L']d^:= J (^ j\(fo, L^) Af^ d{ , (60) 

with E{fo, L^) being the "inverse" of the monotone function fo{E, L^) (the results obtained by the authors quoted above are 
for isotropic systems, but there is no problem to generalize them to the anisotropic case, as we do here). Formally, W2 is easily 
shown to satisfy the relation 

w2[-{h,m] = H'''^\m (61) 

for any function gi . The admissible Q2 is taken to be the set of all the /i satisfying the constraints 

7rd{ = 0and lhC"(fo)di = 0, (62) 



which imply the conservation of mass and _ffc-function to first order. The constraints which are retained from VP are thus 
quite weak, and, as a consequence, it has not been possible up to now to derive in this framework stability results as complete 
as those furnished by the energy and the symplectic methods. 

c) The functional used by (Sygnet et al. 1984) appears to be FF2[/i], as in the previous method, but the set of admissible 
functions fi is not clearly specified. The authors, however, effect without further justifications the change of dependant variable 
fi ^ fi defined by 



h{r,E,L^) 



dr 



{r,E,L^) (63) 



[their Eq. (B3), written with our notations], where use is made of (r, E, L^) as independant variables [actually, this is a proper 
choice only in the region {vr > 0} or in the region {vr < 0}]. But such a change can be valid only for particular perturbations. 
First, it is easy to see that any /i of the form Eq. (63) is symplectic, being generated by 'gi{r, E , L^) := (rvrli}{r, E, L^). 
Secondly, the new function fi introduced by this relation can be regular only if ^ = for rvr = 0. In our work, on the 
contrary, these conditions are explicit part of a coherent formalism: The symplectic nature of the perturbations is a basic 
element of the framework, while the vanishing of 'gi for rVr = is automatically ensured by restricting our attention to the 
antisymmetric part 'gi~ oi'gi, the part of the functional related to turning out to be trivially positive. 



5 STABILITY WITH RESPECT TO ASPHERICAL PERTURBATIONS 
5.1 A further decomposition 

Using the relation 

Sfi = -{fo,8gi} = -foE{E,8gi} - f„L2{L\gi} , (64) 
we can write the quantity H^^\Sgi] as a sum of two terms: 

H'^''>[8g^] = Hi'\8g^] + Hi'\8g^], (65) 
with 
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1 f \{fo,Sgi}\ 



2 



-foE 



{L^8gl}{fo,8g^}dC 



(66) 
(67) 



5.2 Sign of _ffp)[«(/i] 
We now set 

*/i = -{/o,«ffi} =: -/oiS«$i 



(68) 



i.e., we use the standard trick [introduced in the gravitational context by (Lynden-Bell 1966)] which consists to single out the 
"quasi-static part" in the variation 8fi of the distribution function. Then, taking into account the relation 



d{ = -^ /|V,«$i|'dx, 



(69) 



we can write 
K2)r.„,i _ 1 /'Ml 



2 

1 

2 

1 

8^ 



f0E{8<!>lf + («/l +/0i5«$l)«$l 



|V,«$i|' -47rG ( I {-foE)dv]\8<!>if 



2 / -/OS 



Following (Aly & Perez 1992), we now set 
Then 



(70) 



(71) 



1 



2 



*o' (|V,«;|' - 47rG(-/oi5)h|') + \wf{V,^'of + ,\wf] dx 



1 



/o£ ^ SttG 



(*o) - 47rG / (-/ois) dv \w\' - hr^oV^*^, 



dx. 



The quantity V^^'o can be computed by differentiating Eq. (14) with respect to r, which gives 



Vl-^'o = AirGip'o + p',) + 2^1 = 47rG 



+ 2- 



(72) 



(73) 



On the other hand, w has zero average value on any spherical surface of center O, and then, by the so-called Wirtinger 
inequality (Aly & Perez 1992), we have for any value of r 



(|V,,«;|' - ^hl") dQ > 0, 



(74) 



with equality holding if and only if w is of the form w(r, 9, (f)) = wo{r)'x:.a for some constant vector a [in Eq. (74), we have set 
V:,s := V:, - f9/9r and dQ := sin6id6id(/i] . 

Using these results in Eq. (72), we obtain eventually 



1 fiSfi) 



foE ^ SttG 



> 



1 



2 



1 



foE ^ SttG 



(*o)' 



(*0 



dw 



dr 
dw ^ 



dr 



' + - 2^^ + 47rG*i (^(-p^) + 2r J v^-foL^) dv 

+ 47rG*i ({-p'^) + 2r [ «?(-/oi2) dv ) \wf 



dx. 



Therefore, H[^\8gi]>0 for all gi if 

P'e<0, 

and 

foL^ < . 



dx 



(75) 



(76) 



(77) 



Let us now assume that both conditions are satisfied, and that H[^\8gi] = 0. Then: (i) 8~fi = 0; (ii) dw/dr = and 
Wirtinger's inequality reduces to an equality, whence w = x.a. (iii) If either ^ or /oi,2 ^ 0, we must also have w = for 
some values of r. Then w = everywhere, and 8fi = 0. (iv) If pj. = (which implies pe = and $e = 0) and /oi,2 = 0, the 
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previous results and the definition of w imply 8fi = — /os^'oX.a = — a.V^/o, which means that the perturbation just results 
from a rigid translation of the system of a vector a. But there is no such perturbations with a 7^ generated by the elements 
of Q (see §3), and then we have in that case too w = and 8fi = 0. We have thus proven eventually that, under conditions 
(76) and (77), 

H['\8g^]>0 if «/i ^ . (78) 

5.3 Sign of iff 

(2) 

Clearly, the term H2 [8gi] can be of either sign, depending on the choice of the generator. We even know from the general 
theorem of Paper 1 quoted in §3.2, that, by an appropriate choice of gi, it can be made sufficiently negative to overcome 
the positivity of H^^^[gi] + H[^\8gi] which holds under conditions (76) and (77). Then we could only try to classify the 
perturbations according to the sign of that quantity. 

Even that more modest goal appears to be difficult to reach, and we shall content ourselves here with the simple following 
remark: There is a particular class of perturbations, defined by the condition 

foL'AL\gi} = foL'AL\8gi} = 0, (79) 

for which H2^\8gi] vanishes [note the use in Eq. (79) of the obvious relation {L^ ,'gi} = 0]. We feel that these elements of 
Q - which thus commute with in the region of the phase space where /oi,2 7^ - play some role in the dynamics of the 
system, and then they deserve a particular name: We shall call them "preserving generators" - the associated perturbations 
being the "preserving perturbations" -, and denote as V their set. 

Clearly, 'P is a linear subspace of Q, and the general form of its elements [which do automatically satisfy condition (26)] can 
be found as follows. Consider the part Qi of Qo where /oi,2 ^ 0. We first note that there are five obvious solutions to Eq. (79) in 
Qi, namely: r, pr, Lx = — coi 9 cos 4>p^ — sin 4>pg, Ly = — coi 9 sin 4>p^-\- cos 4>pg and = P4>- Moreover, they are linearly inde- 
pendant, which can be most easily checked by considering the Jacobian determinant D{r, pr, L^, Ly, Lz)/ D{t, 9, (f), pr, pe, P4>), 
which turns out to be of rank 5. As the general solution to Eq. (79) in Qi must depend on at most five integrals, we can thus 
conclude that any preserving generator is of the form 

gi{r,9,4>,pr,pe,p,p) = gi {r,pr,P4,, — cot 9 cos 4>p^ — sin 4>pe, — cot 9 sin 4>p^ + cos 4>pe) , (80) 

in Qi, and arbitrary elsewhere. Of course, all the spherically symmetric gi belong to V . But this set contains many other 
elements, as the spherically symmetric elements depend on three variables \gi = ^ = jrj (r, pr, + +-^0)]) while the general 
preserving ones depend on five variables. Unfortunately, V is not "closed under evolution" (see §3.2). Indeed, it results at once 
from the equation of evolution (25) that condition (79) is time-invariant only if the potential $1 stays spherically symmetric, 
which can be the case only for the particular spherically symmetric perturbations. 
We can thus eventually summarize the results of §5.2-5.3 in the form of 

Theorem 2: If the conditions (18), (76) and (77) are satisfied, then we have H^^\8gi] > for all the nontrivial aspherical 
generators of Q satisfying fQj^2{L^ , 8gi} = 0, 

5.4 Consequences for stability 

From Theorem 2 above and the general statements of §3.2, we can first conclude at once that an isotropic system is stable 
with respect to any aspherical perturbations - and then is stable with respect to any perturbation, because of the result of 
§4.2, - if condition (76) is satisfied, i.e., if the external mass density pe is a nonincreasing function of r (which is always the 
case in the applications). 

For anisotropic systems satisfying (76) and (77), our Theorem 2 shows that any nontrivial aspherical perturbation 
generated by a preserving 8gi has positive energy - the same result holding true for any preserving gi because of theorem 1. 
This completely new result is unfortunately more difficult to interpret. Strictly speaking, neither of the general stability criteria 
established thus far do apply here, as the preserving character of a perturbation is conserved in time only for spherically 
symmetric perturbations {{fo,gi} = 0). However, we feel that our result has some relevance for the stability problem. 
Numerical simulations giving some support to this idea are reported in (Perez et al. 1995). 

5.5 Comparison with previous works 

For isotropic systems, the stability result above has already been derived by the thermodynamic method and the energy 
principle ((Antonov 1962), (Lynden-Bell & Sannit 1969), (Binney & Tremaine 1987), where it is referred to as the Antonov- 
Lebowitz theorem) in the case where there is no external potential (i.e., $e = 0). The new proof we have presented here, 
however, is more simple and transparent, and it is self-contained (e.g., it does not use results on the stability of gaseous stars). 
Moreover, it does not appeal to some unproven assumptions on the completeness of the modes of some operator (assumptions 
which may certainly be proven, but whith much efforts). Our work thus answers an important question addressed by (Kandrup 
& Sygnet 1985) about the possibility of a stability proof not using some peculiar completeness theorem, which certainly justified 
to present it in details. 
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Concerning the aspherical stability of an anisotropic system, there seems to be only one published result based on the 
consideration of an energy functional. It is due to (Gillon et al. 1976), and states that any spherically symmetric steady 
state satisfying $e = and conditions (18) and (77) is stable with respect to any aspherical perturbations. There has been 
some doubts repeatedly casted in the literature on the validity of this result (e.g., (Binney & Tremaine 1987)), but no explicit 
rebuttal (or approval!) has been yet presented. Owing to its possible importance, we have thus conducted a detailed analysis of 
Gillon et al.'s complicated proof, in which a spherical system is approximated by a multiple water-bag model. Unfortunately, 
we have found two serious flaws, and our conclusion does confirm the fear of the skeptics: 

a) There is an inconsistency in the calculations, as the conservation law for the basic quantity A [their Eq. (49)] is obtained 
by using both the nonlinear and the linearized VP, without distinguishing carefully the order of the various terms. When the 
derivation is made correctly, it does not seem to lead to an interesting result. 

b) If we forget about this inconsistency and admit for a while the validity of their Eq. (49), then we must note that the 
"conserved" quantity A [their Eq. (52)] takes a positive value Aeq > at equilibrium pust make a±k = in their Eq. (52), 
where a±k are the functions describing an arbitrary perturbation]. Therefore, to be a Lyapunov function, A should satisfy the 
condition A > Aeq for any perturbation rather than the stated one, A > 0. But it is easy to check that the former condition 
is certainly not fulfilled, as A can be made larger or smaller than Aeq by choosing adequately a±k. Then, even under our 
optimistic assumption, no conclusion could be actually drawn from the sign of A. 



6 EXTENSION TO OTHER SYSTEMS 
6.1 Rotating spherical systems 

Consider a spherically symmetric steady state characterized by the function fs{E,L^) and the potential ^'o, and introduce 
an arbitrary function fa{E, , Lz) such that 

h{E, L\L.) := fs{E, i') + f4E, L\L.) > , (81) 
and 

fa{E,L\-L.) = -fa{E,L\L.). (82) 

As noted long ago by (Lynden-Bell 1960), /o is also a steady state distribution function associated to the potential ^'o (the 
mass density corresponding to /o is the same as the one corresponding to fs), and it describes a system which is rotating about 
the 2-axis. We now discuss whether the stability results derived above can be extended to these more general distribution 
functions. 

Proceeding as in §3.3, we obtain after some algebra 
H^'\!li] = -\ J{fs,gT}{E,gT*}di-l j {h,Sg,}{E,8gl} - j {U,lI}{E,8gl} 

with 



^i=K[-{fs,gI}-{fa,f>gi}], (84) 



«$i = K[-{fs,8g^} - ({fa,8g^} - {fa,8g^})]. (85) 

Because of the presence of the nonspherically symmetric term fa in /o, it is then no longer possible to separate H^^\gi\ into 
two terms depending respectively on 'gi and 8gi , and the analysis becomes rather intricate. 

However, there are two simple - but new - results which can be extracted from the expressions above when dfs/dE < 0. 
Consider first a purely spherically symmetric perturbation generated by 'gi. Then 

i7«[3i] = i7«[^] = -i j{f^,gY}{E,gT'}di-^ j IVlT^dx, (86) 

¥7= *'[-{/., F}], (87) 

i.e., we recover the expressions valid for the spherical state fs- It thus results at once from §4 that H^^^\0\ > for any 
nontrivial 'gi, and our rotating state turns out to be stable with respect to all the spherically symmetric perturbations (their 
character being preserved in time). 

Consider now a system with fs = fs{E) which is submitted to some axisymmetric perturbation {d8gi/d(l> = 0) satisfying 
8gi{ — v,f,) = 8gi{v,f,) (a property also conserved in time). Then stability is obtained in that case too. As easily checked indeed, 
the cross term in Eq. (83) [containing both 'gi and 8gi] and the perturbed potential associated to fa vanish, and the results 
of §4-5 imply H'^^^gi] > 0. 
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6.2 Systems with a nondecreasing distribution function 

The stability of a spherical system when condition (18) is not satisfied has been considered numerically by (Henon 1973). 
From an analytical point of view, it appears to be a quite difficult problem, and here, we shall content ourselves to make two 
points. 

a) Consider the case where foE > in the region Qo where /o > 0. Of course, this implies that /o is discontinuous 
on the boundary c)Qo = | -E({) = Eo < 0}, and there is a violation of our smoothness assumption. Consider a spherical 
perturbation generated by 'gi, and assume that {E,'gi} vanishes on d^o- It is easy to check that H^^^\0\ is still given by 
Eq. (39) [the boundary term related to the presence of the delta- function 8(E — i?o) in the derivative foE disappearing], and 
that H^^^\0\ < for any nontrivial 'gi. Then the system appears to be stable with respect to these particular perturbations 
(§3. 2. a) - a conclusion also obtained by (Kandrup & Sygnet 1985) by means of their energy principle. 

b) Consider the case where the system is isotropic [/o = /o(-E)] and foE may have a changing sign in Qo, and let us 
assume that the system admits a neutral aspherical mode generated by 8gi. Then we have 8~fi = and H^^\8gi\ = (Paper 
1), which implies at once, by using the arguments at the end of §5.1, that 8fi = 0. We have thus proven the following new 
result: Whichever be the sign of foE, an isotropic spherically symmetric system cannot admit an aspherical neutral mode. 



7 CONCLUSION 

In this paper, we have considered a self-gravitating coUisionless system, in which the particles are also submitted to an external 
potential $e. We have first presented some new considerations on the relations between the spherical symmetry of a steady 
state and the form of its distribution function /o. In particular, we have argued that, under quite general assumptions, a 
system with an /o depending only on E and is necessarily spherically symmetric. 

We have thus reinvestigated from an analytical point of view the problem of the linear stability of this system in the 
consistent framework of the symplectic method. By effecting a systematic study of the sign of the energy of an arbitrary 
symplectic perturbation, we have reached in particular the following conclusions, valid when foE < 0: 

a) Any steady state is stable with respect to all the spherically symmetric perturbations. 

b) An isotropic system [fo = fo{E)] is also stable with respect to all the aspherical perturbations, at least if the external 
mass density pe is a nonincreasing function of the radial coordinate r. Of course, this result and the one in (a) above are well 
known in the case where $e = 0. But even if we restrict our attention to this particular case, our proofs are much simpler and 
complete than all the previous ones, bringing about some new important elements which were called for by many authors. In 
particular, our proof of the second quoted result shows more clearly the origin of the stability, and it is the first which does 
not appeal to some unproven assumption on the completeness of a set of modes. 

c) For an anisotropic system with /oi,2 < and p'^ < 0, any nontrivial "preserving perturbation" - i.e., one having a 
generator commuting with , which of course does not mean that it is spherically symmetric - has a positive energy. The 
exact meaning of this result is still not completely clear, and we have tried to elucidate it by effecting numerical investigations. 
The results obtained thus far [in particular for the case where the distribution function is of the Ossipkov-Meritt type (see, 
e.g., (Binney & Tremaine 1987)) and there is no external potential] are reported in the companion paper (Perez et al. 1995). 
More general situations, currently under intensive study, will be analyzed in forthcoming papers. 

Finally: 

a) We have analyzed in details a challenged result of (Gillon et al. 1976), according to which stability with respect to any 
perturbation holds if /oi,2 < 0, and found the proof to contain some flaws, that we have clearly explicited. 

b) We have shown that a spherical rotating system is stable with respect to all the spherically symmetric perturbations, 
and [when fs = fs{E)] to all the axisymmetric even ones. 

c) We have proven that an isotropic system with a derivative foE which may change its sign, cannot admit a nontrivial 
neutral mode (i.e., one differing from a mere global translation of the system). 
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